The Simplex and Policy-Iteration Methods Are Strongly Polynomial for the Markov Decision Problem with a Fixed Discount Rate

We prove that the classic policy-iteration method (Howard 1960) and the original simplex method with the most-negative-reduced-cost pivoting rule (Dantzig 1947) are strongly polynomial-time algorithms for solving the Markov decision problem (MDP) with a fixed discount rate. Furthermore, the computational complexity of the policy-iteration and simplex methods is superior to that of the only known strongly polynomial-time interiorpoint algorithm (Ye 2005) for solving this problem. The result is surprising since the simplex method with the same pivoting rule was shown to be exponential for solving a general linear programming (LP) problem (Klee and Minty 1972), the simplex method with the smallest-index pivoting rule was shown to be exponential for solving an MDP regardless of discount rates (Melekopoglou and Condon 1994), and the policy-iteration method was recently shown to be exponential for solving undiscounted MDPs under the average cost criterion. We also extend the result to solving MDPs with transient substochastic transition matrices whose spectral radii uniformly minorize one.